danvi/dim1486/generalize dan perception stack by bogwi · Pull Request #3723 · dimensionalOS/dimos

bogwi · 2026-08-27T11:33:45Z

Generealize perception stack

This PR builds, improves, and adds new functionality based on the foundation of these previous PRs:
#3496
#3422

How to test and what to expect

refference machine:
Lenovo Legion laptop, AMD Ruyzen AI 7, 32GB RAM, NVIDIA RTX 5070 Laptop edition (8 GB)

Caller - operator/agent decide the localization parameters at runtime

every localize call can have own tuning parameters supplied

At the moment we have ROOM policy, and default table policy, think xArm gripper.

This accounts for different environment and rig. We can control a detection score to take only the best candidates, and an acceptance floor, you can control segmentation, etc. Check localize.py

xArm

This uses the deault location for xArm dataset, already on main, xarm6_worldbelief_20260729_203624_161992.db.
You do not need to specify the --dataset <location_to_recording_bag_.db>

time uv run python -m dimos.perception.memory.tool_localize "roll of black tape" "book" "pen" "red marker" "yellow sticky notes" pc_xArm_all_multi.rrd --from 427 --duration 76 --multi && uv run dimos-viewer pc_xArm_all_multi.rrd

# cold start, weights ~ 11s reff.machine
real	0m34.002s 
user	1m4.301s
sys	    0m5.702s

G1

You will need a g1 recording for this, check this PR, it's on main already, #3527

time uv run python -m dimos.perception.memory.tool_localize "paper box" "plant in a pot" "shoes" "wooden crate" "office chair" "bag" g1_sf_120_180_localize.rrd --dataset data/g1_sf_office.db --from 120 --duration 60 --multi && uv run dimos-viewer g1_sf_120_180_localize.rrd

# cold start, weights ~ 11s reff.machine
real	1m11.971s
user	4m44.586s
sys	    0m11.864s

Go2

Run on the default go2_short.db

time uv run python -m dimos.perception.memory.tool_localize "table" "white robot" "potted plants" "wooden doors" "coke bottle" "fanta bottle" "sprite bottle" "lays chips" "pillow" "green plant" "plant pot" go2_short_00_60.rrd --dataset data/go2_short.db --from 0 --duration 60 --multi && uv run dimos-viewer go2_short_00_60.rrd

# cold start, weights ~ 11s reff.machine
real    1m4.001s
user    1m45.383s
sys     0m24.279s

A live blueprint addition showcasing the localize api on the live continuous stream.

The examples streams over go2_short.db recording that comes with dimos. Queries can be executed in real-time from dimos shell or mcp calls.

Check dedicated readme for guide how to run it, dimos/dimos/perception/memory/blueprints/README.md

Important

Performance is affected:

by a cold run, weights need to be loaded, around ~11 seconds on my machine
saving to *.rrd format

tool_inventory.py has been not touched by this commit, will come later.
Use localize() API for testing and building
Re-id and spatial memory are underway too
Improving objects 2D into 3D is also underway
supporting mid360 lidar is also undeerway
Any questions - contact me directly, @bogwi

…he window's cell range - one fancy-index lookup per snapshot's points and one rectangle memset after it

codecov · 2026-08-27T11:42:37Z

Codecov Report

❌ Patch coverage is 26.43098% with 874 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/perception/memory/rig.py	16.37%	429 Missing ⚠️
dimos/perception/memory/localize.py	15.16%	207 Missing ⚠️
.../perception/memory/blueprints/go2_localize_live.py	36.01%	167 Missing ⚠️
...erception/detection/type/detection3d/pointcloud.py	54.32%	28 Missing and 9 partials ⚠️
...n/detection/type/detection3d/pointcloud_filters.py	20.00%	16 Missing ⚠️
dimos/perception/memory/support_plane.py	35.71%	9 Missing ⚠️
dimos/perception/memory/dandetect.py	55.55%	4 Missing ⚠️
dimos/memory/backend.py	66.66%	1 Missing and 1 partial ⚠️
.../detection/type/detection3d/imageDetections3DPC.py	33.33%	2 Missing ⚠️
dimos/memory/stream.py	0.00%	1 Missing ⚠️

@@            Coverage Diff             @@
##             main    #3723      +/-   ##
==========================================
- Coverage   77.46%   77.41%   -0.05%     
==========================================
  Files        1302     1345      +43     
  Lines      123061   129215    +6154     
  Branches    10772    11924    +1152     
==========================================
+ Hits        95330   100038    +4708     
- Misses      24924    25993    +1069     
- Partials     2807     3184     +377

Flag	Coverage Δ
OS-ubuntu-24.04-arm	`74.08% <23.06%> (+0.59%)`	⬆️
OS-ubuntu-latest	`74.60% <23.06%> (+0.55%)`	⬆️
Py-3.10	`74.60% <23.06%> (+0.55%)`	⬆️
Py-3.11	`74.60% <23.06%> (+0.55%)`	⬆️
Py-3.12	`74.60% <23.06%> (+0.55%)`	⬆️
Py-3.13	`74.60% <23.06%> (+0.54%)`	⬆️
Py-3.14	`74.61% <23.06%> (+0.55%)`	⬆️
Py-3.14t	`74.60% <23.06%> (+0.55%)`	⬆️
SelfHosted-Large	`30.63% <13.38%> (+0.29%)`	⬆️
SelfHosted-Linux	`34.74% <44.88%> (-0.12%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/agents/mcp/mcp_adapter.py	`71.60% <100.00%> (-0.35%)`	⬇️
dimos/cli/commands/mcp.py	`63.63% <100.00%> (ø)`
dimos/core/global_config.py	`86.91% <100.00%> (+0.77%)`	⬆️
dimos/memory/utils/sqlite.py	`100.00% <100.00%> (ø)`
...tion/detection/type/detection3d/test_pointcloud.py	`100.00% <100.00%> (ø)`
dimos/perception/memory/types.py	`100.00% <100.00%> (ø)`
dimos/robot/all_blueprints.py	`100.00% <ø> (ø)`
dimos/memory/stream.py	`83.48% <0.00%> (ø)`
dimos/memory/backend.py	`86.18% <66.66%> (+0.94%)`	⬆️
.../detection/type/detection3d/imageDetections3DPC.py	`60.00% <33.33%> (-9.24%)`	⬇️
... and 7 more

... and 201 files with indirect coverage changes

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

leshy

basically escaped mem2 immediately, first comments first view to push back on this, might have more stuff

test_detection3dpc pinned obb.center and obb.extent to values from when from_2d projected through a bare pinhole matrix and ignored camera_info.D. The camera under test is go2_front_camera_720p, declared equidistant. project_pixels now applies that model, shifting projected pixels by a median of 20 px and moving the suitcase OBB center 18 mm in y, just past the 0.1 window. Tolerances and the file's other assertions are unchanged.

…ocalize call in runtime

leshy

didn't review all, we'll need to meet and reorganize

leshy · 2026-09-02T13:54:56Z

+        stronger = self if self.confidence >= other.confidence else other
+        return Detection3DPC(
+            image=later.image,
+            bbox=later.bbox,


you should expand bbox, not take latest one, your result is a union now

Do not agree, expanind is not a fix because two boxes are not in the same image. add` keeps image, bbox and transform from one and the same sighting so the union stays a valid 2D detection: the box is where the object appears in that image.

This related to a later comment. The real fix is to drop the sum

leshy · 2026-09-02T14:44:34Z

+
+        span = _camera_span(rig)
+        rig.mobile = span > MOBILE_SPAN_M
+        if rig.mobile:


I don't think we should have these meta-settings, below is already exposed to the config

Yes, agreed

leshy · 2026-09-02T15:19:37Z

+        candidates = list(self.poses.at(ts, self.tf_tolerance))
+        if not candidates:
+            return None
+        earlier = [o for o in candidates if o.ts <= ts]


this whole function is pretty complex, there must be an easier way to find a pose at some time? if we need interpolation we should generalize somewhere

leshy · 2026-09-02T15:20:17Z

+
+    def world_to_optical(self, ts: float) -> Transform | None:
+        ts -= self.color_delay  # color stamps lag; the capture instant is earlier
+        if self.tf is not None:


should probably just insist on valid tf to not deal with special cases

leshy · 2026-09-02T15:33:44Z

most of these function descriptions seem ai generated and very difficult to understand

spomichter · 2026-09-04T20:51:02Z

+    # Seconds an MCP client waits for a tool to answer. A skill that thinks
+    # for longer than this is cut off at the client, not the server, so the
+    # caller owns the number.
+    mcp_timeout: int = 30


spomichter · 2026-09-04T20:51:21Z


 logger = setup_logger()

-DEFAULT_TIMEOUT = 30


yes was bad thanks. no globals no in globalconfig

spomichter · 2026-09-04T20:52:06Z

@@ -27,6 +27,9 @@ def open_sqlite_connection(path: str | Path) -> sqlite3.Connection:
    conn = sqlite3.connect(path, check_same_thread=False)
    conn.execute("PRAGMA journal_mode=WAL")
    conn.execute("PRAGMA synchronous=NORMAL")
+    # WAL has one writer at a time; concurrent writer threads (a recorder


prefer limited comments. i put in my claude.md to super limit ai comments

spomichter · 2026-09-04T20:53:18Z

+        """
+        return self.subscribe(
+            lambda _: None,
+            on_error=lambda e: logger.error("drain_thread() pipeline died: %s", e, exc_info=e),


what i personally do if i have refactor code unrealted to the PR -- i PR it seperatley to keep main feature PR clean

spomichter · 2026-09-04T20:53:57Z

@@ -27,9 +28,14 @@
 from dimos.perception.detection.type.detection2d.bbox import Detection2DBBox
 from dimos.perception.detection.type.detection2d.imageDetections2D import ImageDetections2D

+# ~3.8 MB of GPU memory per cached frame
+_FEATURE_CACHE_MAX = 128


maybe this should be global constant in constants.py but idk enough about this model

spomichter · 2026-09-04T20:54:11Z


 class Owlv2Config(HuggingFaceModelConfig):
    model_name: str = "google/owlv2-base-patch16-ensemble"
+    # float16 runs the forward under autocast at roughly half the latency;


delete comment

bogwi added 15 commits August 27, 2026 20:20

init generalizing refactor

e2cbb3e

improve perception stack; test across different sourced

32e9763

refactor localize to memory transforms

35c65c8

impl: (a) All instances per query. (b) Global point cloud per instance

37d22ae

filter before decode in _vector_search

eef787d

support plane cached per pose cell

fe42124

OWLv2 score-row cache keyed by (frame ts, label), batched misses

840463a

add persistent identity groups, identity_store.py

131e03d

make fuse to compute the exact all-time mean

909e6ac

detection small fixes

e717a42

project images; tool_localize update

04706c5

improve detection

429c8a8

in clout_at: replace the broadcast with a boolean covered-grid over t…

6935a68

…he window's cell range - one fancy-index lookup per snapshot's points and one rectangle memset after it

smal improv to lattice_quantum foo

26e993f

ship go2_short.db.rig.json inside the go2_short LFS archive

ff031de

bogwi requested review from Dreamsorcerer, aclauer, leshy, mustafab0, paul-nechifor and spomichter as code owners August 27, 2026 11:33

[autofix.ci] apply automated fixes

8a7e58c

This comment was marked as outdated.

Sign in to view

greptile-apps Bot reviewed Aug 27, 2026

View reviewed changes

Comment thread dimos/perception/memory/localize.py Outdated

Comment thread dimos/perception/memory/localize.py Outdated

leshy requested changes Aug 27, 2026

View reviewed changes

Comment thread dimos/perception/memory/localize.py Outdated

Comment thread dimos/perception/memory/localize.py Outdated

Comment thread dimos/perception/memory/localize.py Outdated

Comment thread dimos/perception/memory/localize.py Outdated

Comment thread dimos/perception/memory/localize.py

bogwi and others added 4 commits August 28, 2026 11:27

fix mypy

3d151d5

move to mem api

55355cd

[autofix.ci] apply automated fixes

533e0bc

bogwi requested a review from leshy August 30, 2026 12:57

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 30, 2026

add LocalizePolicy for the localize caller. The caller can tune any l…

96ce3a6

…ocalize call in runtime

github-actions Bot added ready-to-merge Required CI checks have passed on this PR and removed ready-to-merge Required CI checks have passed on this PR labels Aug 31, 2026

enhanse rig.py

57d11cb

github-actions Bot added ready-to-merge Required CI checks have passed on this PR and removed ready-to-merge Required CI checks have passed on this PR labels Aug 31, 2026

bogwi added 3 commits September 1, 2026 12:07

add localize live blueprint

b32dd57

profile memory for rerun

3e10486

make mcp call caller dependent; preserve the default 30 sec rule

886adbb

github-actions Bot added ready-to-merge Required CI checks have passed on this PR and removed ready-to-merge Required CI checks have passed on this PR labels Sep 1, 2026

leshy requested changes Sep 2, 2026

View reviewed changes

bogwi marked this pull request as draft September 3, 2026 06:55

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Sep 3, 2026

bogwi added 3 commits September 3, 2026 18:00

remove inventory api

fcd5b70

refactor the stack

de79d19

fix mypy

00f0c66

bogwi force-pushed the danvi/dim1486/generalize-dan-perception-stack branch from da6b87c to 00f0c66 Compare September 3, 2026 17:17

spomichter reviewed Sep 4, 2026

View reviewed changes

bogwi closed this Sep 5, 2026

bogwi reopened this Sep 5, 2026

Uh oh!

Conversation

bogwi commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Generealize perception stack

How to test and what to expect

Caller - operator/agent decide the localization parameters at runtime

xArm

G1

Go2

A live blueprint addition showcasing the localize api on the live continuous stream.

Important

Uh oh!

codecov Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

leshy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leshy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leshy commented Sep 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bogwi commented Aug 27, 2026 •

edited

Loading

codecov Bot commented Aug 27, 2026 •

edited

Loading

leshy left a comment •

edited

Loading